Papers with Large-scale vision language
PuMer: Pruning and Merging Tokens for Efficient Vision Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Large-scale vision language models use Transformers to perform cross-modal interactions . state-of-the-art models are memory intensive and expensive due to quadratic complexity . |
| Approach: | They propose a token reduction framework that uses text-informed Pruning and modality-aware Merging strategies to progressively reduce the tokens of input image and text. |
| Outcome: | The proposed framework improves inference speed and memory footprint on four vision language tasks. |